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Charter Schools’ Performance and Accountability: 

A Disconnect 

Gerald W. Bracey 
George Mason University 

Executive Summary 

This report argues that evidence exists for the case that the charter school 
movement is largely a failed reform. The report puts the charter school movement in the 
context of dissatisfaction with public schools and the public sector in general. It then 
describes the claims for charters made by the early charter school advocates, emphasizing 
the advocates’ promise of increased achievement. From there, the report reviews 
evaluations of charter schools in Arizona, California, Michigan, Ohio, Illinois, North 
Carolina, and Texas, as well as several national evaluations. 

The review shows that charters have not lived up to their promise of increased 
achievement. This failure is surprising given that charter schools are small (most have 
fewer than 200 students) with small classes, two factors known to increase achievement. 
This failure becomes even harder to understand given the advantages that charters enjoy 
in their freedom from the rules, regulations, and contracts that are said to bureaucratically 
burden the public schools. 

It appears that charter school advocates who believed that charters could increase 
achievement and should be held accountable for doing so have lost control of the pro- 
charter movement to those for whom deregulation is a sufficient condition for declaring 



success. 
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Why Charter Schools and Where They Came From 

Before exploring accountability in charters, it is necessary to examine the origins 
of charters and why people were interested in charters in the first place. Michael 
Mintrom and David Plank provide a useful summary of the trends and contexts that made 
people receptive to charter schools: 

Citizens’ confidence in the efficiency and competence of public sector 
bureaucracies has declined, and markets and quasi-markets have had to fulfill a 
variety of public purposes. Among other things, this has entailed a move away 
from the direct provision of public services by government agencies toward 
increased reliance on subsidies, incentives, and accountability frameworks as 
policy instruments to encourage private actors to accomplish policy goals. 

The acceleration of these trends has coincided with a rise in public 
dissatisfaction with the traditional public school system. This originates both in 
the failure of public school systems to address the educational necessities of poor 
children, including especially urban and minority children, and in the schools’ 
perceived failures to provide students with the academic skills that they will need 
to compete in a global economy with better-schooled Asians and Europeans . 1 




Joseph Murphy and Catherine Dunn Shiffman documented well the numerous 



social commentators who decried the role of government. Citing about a dozen such 
commentators, Murphy and Shiffman conclude: “These analysts paint a picture of 

political bankruptcy, a vaguely defined state of popular alienation, and disaffection from 

2 

government which stops short of revolution.” 

Charter schools were conceptually invented in the 1970’s by a Massachusetts 
teacher, Ray Budde, but the idea received little attention for more than a decade. They 
got their largest boost in 1988 from Albert Shanker, president of the American Federation 
of Teachers. Shanker certainly agreed about the failure of the schools. He later changed 
his mind about the value of charters. Shanker came to feel that charters were “at best a 
partial answer to the problems that afflict our schools” and, worse, “the basic premise of 
charter schools ensures that whatever common ground schools now share will 
disappear.” 4 

But about the conventional public schools’ low quality, Shanker had no doubt. 

He opened three consecutive columns in the New York Times with these sentences: 

• The achievement of U. S. students in grades K-12 is very poor. 5 

• American students are performing at much lower levels than students in other 
industrialized nations. 6 

• International examinations designed to compare students from all over the 

' 7 

world usually show American students at or near the bottom. 

Subsequent international studies have shown that none of these statements was 
true, but they were, and are, widely believed. In May, 2003, Daniel Henninger, an editor 
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the editorial pages of the Wall Street Journal, asserted: “Americans are almost 

Q 

unanimous, public schools are awful.” 

The events of September 11, 2001, and the scandals in corporate America have 
led to a rethinking of the uses of government and of the so-called efficiency of the private 
sector. Not much of that rethinking, however, seems to have yet permeated discussions 
about public schools. Indeed, a 2003 report from People for the American Way 
documented $75 mi llion in U.S. Department of Education funds allocated to groups with 
a privatization agenda, 9 and the Washington Post produced an article in 2004 describing 
how the Education Department favors the political right. 10 

Charter schools did not become popular only with those who favored 
privatization. A number of charter school advocates could be traced back to the reform 
movements of the late 1960’s and 1970’s, movements that emanated mostly from liberals 
and the left, such as the attempts to establish “free schools” and “alternative schools.” 
Whatever their origin, charter schools arrived promising accountability. A 1996 
statement from Joe Nathan, himself a veteran of the alternative schools movement, was 
typical: 

Hundreds of charter schools have been created around this nation by educators 
who are willing to put their jobs on the line to say, “If we can’t improve student 
achievement, close down our school. This is accountability — clear specific and 
real. 11 

This might be called “The Nathan Criterion” and used as a basis for evaluating 
charter performance. Nathan expanded on this anthem and also provided an excellent 
description of the very concept of a charter school: 
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The charter school idea is about the creation of more accountable public schools 
and the removal of the “exclusive franchise” that local school boards presently 
have. Charter schools are public, nonsectarian schools that do not have 
admissions tests but that operate under a written contract, or charter , from a 
school board or some other organization such as a state school board. These 
contracts specify how the school will be held accountable for improved student 
achievement, in exchange for a waiver of most rules and regulations governing 
how they operate. Charter schools that improve achievement have their contracts 
renewed. Charter schools that do not improve student achievement over the 
contract’s period are closed. . . 

The charter idea is not just about the creation of new, more accountable 

public schools or the conversion of existing public schools. The charter idea also 

12 

introduces fair, thoughtful competition into public education. 

A year later, Bruno Manno, Chester E. Finn, Jr., Lou Ann Bierlein, and Gregg 
Vanourek also emphasized the importance of accountability: 

Not only will it (accountability) make or break the charter “movement” itself, it 
will also be a primary source of evidence as to whether that movement is making 
a valuable contribution to the improved education of American children. . .or is 

1 T 

another half tried reform fad that sinks into the sand like so many others. 

The most explicit and ambitious statement of what charter schools would 
accomplish came from Ted Kolderie, who emphasized the systemic aspect: 

Too often those asking “What’s happening?” (in regard to the impact of a charter 
law) look only at the schools created and the students enrolled: the first order 
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effects of a law. There are also second-order effects: changes/responses in the 
mainline system when laws are enacted and schools created. An evaluation needs 
to look for these. . .Despite what the words seem to imply, “charter schools” is not 
basically about the schools. For the teachers who found them and the students 
who enroll them, true, it is the schools that are important. But for others, from the 
beginning, “charter schools” has been about the system-reform, a way for the state 
to cause the system to improve. The schools are instrumental. 14 

More recently, in their introduction to their study of charter school accountability, 
Hill et al. reiterated the performance vs. compliance criterion forjudging charters: 

Conventional public schools are considered accountable because they must follow 
all the rules set by local and state school boards, and abide by all the provisions of 
contracts that these boards enter with unions and other organizations. Charter 
schools are exempted from many of these rules, and instead are required to 
demonstrate student learning. 15 

At the time the charter movement was gathering momentum, some researchers 
expressed doubt that such accountability would ever come to pass. In the same year that 
Nathan delivered his credo, Alex Molnar (then of the University of Wisconsin- 
Milwaukee, now at Arizona State University) weighed in with skepticism: 

Charter schools will fail, fraud will be uncovered, and tax dollars will be wasted. 
But just as certainly, glowing testimony will be paid to the dedication and 
sacrifice of the selfless teachers and administrators at some “Chartermetoo” 
school who transformed the lives of their students and proved the success of 
charter school reform. Free-market zealots will either claim vindication or argue 

Page 5 of 61 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0505-113-EPRU.pdf 





that their revolutionary ideas need more time to work. Supporters of public 
education will call the experiment a costly failure and marvel at the willingness to 
spend large sums on unproven alternatives while cutting resources for the public 
system that serves most children. With an absence of any uniform standards, the 
war of educational anecdotes and misleading statistics will remain “subject to 
interpretation.” And all the while, the desperation of America’s poorest children 
and their families will grow. 

The struggle is not, at its root, between market-based reform and the 
educational status quo. Rather, it is a battle over whether the democratic ideal of 
the common good can survive the onslaught of a market mentality that threatens 
to turn every human relationship, inside and outside the classroom, into a 
commercial transaction. 16 

Two years earlier, Jeffrey Henig of Columbia University, had taken a similar, if 
more subdued position: 

Distressingly, however, the movements for charters, contracting out and privately 
sponsored vouchers so far show few signs of the kind of careful deliberation, 
pragmatic selectivity, or interest in systematic empirical research that ultimately is 
needed if we are going to be able to separate bold claim from proven 
performance. Premature claims of success, reliance on anecdotal and unreliable 
evidence, and will disregard of possibly undesirable consequences are still the 
rule of the day among the vocal advocates. The state and local leaders who are 
pushing most aggressively for vouchers, public school choice, charters, and 
contracting have not made it a priority to link their initiatives to requirements for 
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evaluation, public dissemination of data, and staged implementation. In an 
environment of low information and exaggerated claims, the risks of doing 
damage are as important to consider as the problematic advantages of precipitous 
and undigested reform. 17 

Ten years after Henig’s statement, such claims for charters from their most 
passionate advocates still exist. In late 2003, Jeanne Allen at the Center for Educational 
Reform alleged that 88 major studies, out of 98 total, had found in favor of charter versus 
regular public schools. Allen, however, did not separate the studies in terms of the 
criteria for judging charter schools better, nor by the quality of the research. Some were, 
at best, informal accounts by persons with vested interests in having charters appear 
successful. 

If charter schools had been strictly evaluated by the Nathan Criterion, one can 
speculate it’s hard not to speculate how many would remain open. As it stands, only a 
tiny percentage of charters have been shut down, and those overwhelmingly had their 
charters terminated because they misspent the money, not because they failed to educate 
their students. Otherwise, as a study for the U. S. Department of Education put it, 
“Charter schools rarely face sanctions (revocation or nonrenewal). Furthermore, 
authorizing bodies impose sanctions on charter schools because of sanctions related to 
compliance with regulations and school finances rather than student performance ” 19 
(emphases in the original). Camoy et al. summarized the results on closures in the ten 
years that charter schools had existed as reported by the Center for Educational Reform in 
2002: 21 

• 58 closures for financial reasons 
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• 52 closures for mismanagement reasons 

• 8 closures for district reasons (e.g., state or district budget crisis forcing 
closure of both regular and charter schools) 

• 18 closures for facilities reasons (e.g., unable to find or keep an adequate 
building) 

• 4 closures for “other” reasons 

• 14 closures for academic reasons 

At the time the center counted 2,874 charter schools, meaning that less than one 
half of one percent had been closed for academic reasons. 

Four options offer themselves as explanations of the failure of accountability in 
charters: 

• Charter operators and advocates continued to show no enthusiasm for proper 
evaluation. 

• The concept of accountability has, in some instances, proved to be both more 
complicated and elusive than presented by early enthusiasts. 

• Charter schools have come to be evaluated by a lower standard of 
accountability than are the regular public schools. 

• In some places, charter school operators have conducted something of a bait- 
and-switch: Having promised improved achievement, they later claimed that 
the mere existence of choice was sufficient to justify the continuation of the 
schools. 
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The Limits of Measuring Success by Parental Satisfaction 



Charter supporters have turned increasingly to parental satisfaction as a 
justification for continued support of charter schools. The satisfaction data is suspect, 
though, because it rests solely on simple attitude surveys. The problem with such surveys 
is that they do not take into account the impact of cognitive dissonance. 

The theory of cognitive dissonance was formally put forward in 1957 by 
psychologist Leon Festinger. Festinger held that the very act of choosing created 
cognitive dissonance. While the dissonance would be greatest in choosing among the 
lesser of two evils, dissonance would occur even when choosing among positive 
alternatives. Once a choice is made people avoid and resist information that might 
induce the conclusion that the choice had been a bad one. They seek information that 
affirms that they made the right decision. Thus, in a positive-positive situation, a person 
who chooses a Jaguar over and equally expensive Mercedes will tend to read Jaguar ads 
and avoid positive information about Mercedes. 

In the case of charters, the parents have chosen the schools. For them to then hold 
the idea that they have relegated their children to a bad school or even that they are less 
than fully satisfied with the chosen school would create considerable dissonance. It 
would be difficult for a parent to contend with the internal conflict that would be created 
by the thoughts, “This is a bad school and I have chosen to send my children to this 
school.” For parents who use public schools, no choice is involved (unless the parents 
have chosen to move into the neighborhood served by the school and even in this case the 
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choice is remote from the school itself). It is therefore much easier for parents to criticize 
the teachers, principal, curriculum or whatever. 

Until satisfaction surveys attempt to deal with cognitive dissonance, little can be 
concluded from them and little stock can be put in them. 

Achievement in Charter Schools 

Before applying the Nathan Criterion, it is necessary to examine the charter 
evaluation studies both nationally and in the various states. 

Arizona 

What is clear from the Arizona and Michigan experiences with charters is that 

23 

without rigorous accountability, both students and taxpayers suffer. 

Charter advocates refer to Arizona’s 1994 charter school law as the nation’s 
“strongest.” Toch and critics might call it the most dangerous, because it places the 
fewest accountability requirements on charters and grants charters for the longest time, 

15 years, of any state. According to Toch, one legislator said, “Anyone who could stand 
up and breathe got a charter.” 

Under such circumstances, it is not surprising that the Arizona Department of 
Education began receiving numerous reports about charter school problems: Faculty 
were being paid bonuses for enrollment increases; some high schools were using Arizona 
law that specifies only four hours are required for high school education to conduct three 
shifts of students a day. Toch visited a course called “American Literature Through 
Cinema.” The 30 students in this course were studying The Last of the Mohicans. The 
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students, though, did not see the film; they merely listened to the soundtrack — except, 
notes Toch, those who were asleep. 

The Arizona Department of Education launched an investigation in 1997. The 
evaluation team was “supposed to focus on whether schools were complying with laws 
and charters, but those with education backgrounds found it hard to ignore such things as 
out-of-control classrooms or the absence of instruction.” When the report was 
delivered to State Superintendent Lisa Graham Keegan, Keegan withheld it for a year, 
releasing it only under threat of suit. Before releasing the report, Keegan had her top aide 
remove the evaluators’ hand written notes, permissible under Arizona law. Paul Street, 
one of the evaluators, turned in all of his notes, but when he later checked the 
department’s folder marked “Street,” he found a single scrap of paper. After the report, 
Keegan shut down the evaluation team and transferred the team’s head, Associate 
Superintendent Marilynn Henley to another position. Henley entered the private sector. 
Keegan declared, “In the main, I’m pleased, far and away with the quality of the public 
charter schools.” For her part, Henley said, “I resent [Keegan] and others who say 
charter schools are just like or no worse than district schools. Charter schools are way 
worse.” 25 

A 1999 report from the Morrison Institute for Public Policy at Arizona State 
University observed that 

Improving student achievement is a primary goal of Arizona’s charter school law. 
The issues of accountability and student achievement go hand-in-hand, of course, 
because the amount of improvement in student achievement is often considered to 
be the key indicator of a school’s accountability. As yet, however, there have 
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been no carefully designed general analyses or school-by-school analyses of 
student achievement in charter schools. Indeed, highly credible comparisons of 
achievement between charter school students and regular public school students 
are difficult and costly to perform for several reasons.26 

The Morrison Institute’s own analysis concluded that 

Students in charter schools for two years have increased in achievement (in 
reading, language and mathematics) at approximately similar levels as students at 
regular public schools.... The analysis also revealed that by middle school, 
students who attended charter schools for one year or more began to lag behind 
their regular public school age-mates. By high schools, this effect was even more 
dramatic, with charter school students often 10 to 15 points behind regular public 

27 

school students." 

The analysis did not permit a determination of whether or not charters were 
selecting lower-achieving students or not teaching them well. Toch’s article would 
suggest both: Noting that nearly half of Arizona’s charters at the time were high schools 
run by Educational Management Organizations, Toch wrote, “They target kids on the 
margins of traditional public schools — low achievers, discipline problems, truants — with 
pledges of swift and simple routes to graduation. And many of the companies increase 
their revenues by running two or three four-hour sessions a day and substituting self- 
paced computer instruction for a regular teaching staff.” 28 

In 2001, The Center for Market-Based Education at the Goldwater Institute in 
Phoenix released a study purporting to show larger test score gains in reading for students 
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who stay in charter schools two or three years compared to those who remained in 
traditional public schools (TPSs) for two or three years. In mathematics there were no 
differences, but the report claimed an advantage for charters in reading: 

Our main results for reading are that students enrolled in charter schools for two 
or three consecutive years have an advantage over students staying in TPSs for 
the same periods of time. Students who enrolled in charter schools for two 
consecutive years show a 2.35-2.44 extra point advantage over students who 
stayed in TPSs for two consecutive years. Similarly, students in charter schools 
for three consecutive years show an additional 1.31 extra point advantage over 
students in TPSs for three consecutive years. Both differences are statistically 
significant and also imply significant effect sizes given a (within sample) standard 
deviation compared to those in education production function studies.” 

The test scores are from the SAT 9 and the “points” are percentile ranks. One can 
certainly wonder if a four percentile rank gain in three years means anything in terms of 
practical rather than statistical significance. It would likely correspond to answering one 
or two more items correctly. 

It is quite difficult, however, to know what the researchers actually did. While the 
study was lauded because, at the time, it was the only one to track the same students over 
the period of the study (a state data-capture capability that then only Arizona possessed), 
it is hard to know what analyses the researchers actually performed. Gene V Glass, 
associate dean of research at Arizona State University, indicated that Arizona State 
researchers “who have spent time with it still don’t feel they understand what was 
done.” Similarly, Douglas Harris, an economist at the Economic Policy Institute said, 
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“It’s hard to tell what they actually did. Their descriptions of the methods are really 

O 1 

poor.” Finally, Christopher Nelson of Western Michigan and Kevin Hollenbeck of the 
W. E. Upjohn Institute for Employment Research critiqued the study on several 
methodological grounds, although their report uses many conditional phrases such as “As 
we understand it,” and “it appears as if,” indicating that they, too, are not certain about 
the details of the methodology/ 

In addition, the gains, even if they are actual, are quite small and, in fact, leave the 
charter students still scoring below the TPS students at the end of three years. The 
authors attempt to address this problem, but their exposition is difficult. Finally, the 
authors’ interpretation of “significant effect sizes” is subject to debate. Judgments about 
what constitute small or large effect sizes are partly subjective, but most researchers favor 
an effect size of at least +.20 as having practical significance. According to Harris, if we 
take the data at face value, the effect sizes are about +.10. 

The Goldwater researchers also studied the impact of moving from one type of 
school to another, and the results from these transitional studies do not favor charters: 

Other than being in charter schools for two consecutive years, starting in a charter 
then moving to a TPS is preferred to two years in a TPS which does more for test 
score gain than does starting in a TPS and moving to a charter school. There may 
be several explanations for this. One is that the grounding for learning provided 
in the charter school carries over when one moves to a TPS, whereas a move from 
a TPS to a charter is more disruptive. Whatever deficiencies a student brings with 
him upon enrolling in a charter school may be remediated by the charter schools, 
thus preparing the student to perform better when returning to the TPS. 
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Since the gains in question are one or two percentile ranks, to speak of 
“grounding for learning,” does not seem psychologically or pedagogically sound. An 
equally plausible explanation would be that the students felt more comfortable when they 
returned to a familiar environment, their old school and their old friends. 

Michael Martin, a research analyst for the Arizona School Boards Association, 
analyzed the Goldwater data in some detail and concluded that the gains for students in 
charter schools were illusory. A number of the charters had been private schools of some 
affluence that converted to charter status. The students who had attended these schools 
when they were private schools stayed in the schools. Some 43 percent of charter school 
students who had previously been in public schools returned to the public schools. They 
had not fared well in the charters, but enjoyed a large gain the year they spent in public 
schools after attending charters. Martin argues that this gain, which the Goldwater 
Institute claims as due to “grounding for learning” in charter schools, is more likely a 
“recovery” from the achievement-depressing charter school impact. The apparent gains 
for students who remain in charters are spurious, deriving from the attrition: lower 
achieving students left the charter schools, raising the charter schools’ average test 

34 

scores. 



California 

California’s charter law from 1992 is the second oldest in the country and served 
as the model for many other states. A number of other states’ laws are virtually verbatim 
copies of California’s. California’s law lists as its first purpose, “Improve pupil 
learning.” 
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The first evaluation of California’s charters came in 1997 and had little to say 
about student outcomes: 

Most of our comparisons of charter and noncharter student outcomes yielded 
inconclusive results. Comparable student outcome data between charter schools 
and the non-charter schools in their sponsoring district were hard to find in most 
cases. Even when we did find such data, the data did not allow us to determine 
the relative academic performance of noncharter schools. There were many 
explanations for these circumstances: the absence of a statewide assessment 
system during the life cycles of the charter schools that are currently operating, 
varying philosophies about and approaches to measuring students’ achievement, 
and the lack of consistent approaches by sponsoring agencies in monitoring 
student outcomes at charter schools. 35 

Researchers from WestEd encountered similar assessment problems. Using data 
from one large district, the WestEd team reported inconclusive results: “Charter schools 
maintain or slightly improve their performance over time with respect to students in a 
comparison group of non-charter schools, with a few exceptions.” 36 

A study by Amy Stuart Wells and a team of researchers at UCLA also occurred 
during this period when California’s assessment program was undergoing transitions, 
making test comparisons impossible. The UCLA researchers, though, approached their 
research with explicit tests about claims made for charter schools: they will be more 
accountable, more efficient, will create competition and thereby force the regular public 
schools to change, will create innovative programs and thereby offer more choice, and 
will be more autonomous. The efficiency claim did not hold up. Charter schools 
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required more resources. Successful ones had a well-connected leader who was able to 
tap into private sources of income. 

Competition between the charters and the public schools failed to develop mostly 
because public school officials felt that the charters had unfair advantages. Their ability 
to require parent contracts, or a certain number of hours of parent involvement, their 
ability to select students and limit enrollments, their greater autonomy from regulations 
and their less paperwork meant to the public school people that the playing field wasn’t 
level. Charter schools did not have to accept children after the school year started, giving 
them more stable enrollments. Not having to adjust class activities to the arrival of new 
students or attend to the needs of new students gave them a considerable advantage. 

Wells’ finding corroborated an earlier study by Rofes that included California 
charters. Rofes found that most teachers and administrators at regular schools did not 
view competition from charters in a positive light. According to one teacher, “I don’t do 
my job based on thinking I have a competitor. I do it based on knowing what the child 
needs to grow and have a good education to get somewhere.” 38 Indeed, Rofes’s major 
conclusion was 

Certain innovations and changes in school districts and traditional public 
schools [that Rofes hypothesized would take place] had rarely occurred: 

Few superintendents, principals and teachers in district schools were 
thinking of charter schools as educational laboratories or attempting to 
transfer pedagogical innovations from charters to the district schools. 

Wells found some innovations in areas such as theme-based curricula, but did not 
find, as Kolderie would have hoped, that the charters were serving as “laboratories of 
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innovation” that led to system- wide changes. Instead, she found that ‘‘there are no 
mechanisms in place for charter schools and regular schools to learn from each other.” 

As a consequence, “all but two of the public school educators we interviewed reported 
that they had very little information about what was going on in the charter schools, and 
nearly all of the educators we interviewed said they saw little if any direct impact of 
charter schools on their schools.” 40 

Similarly, few charter operators saw informing public school colleagues about 
their charter schools as part of their purpose. And even if they did, Wells observes, most 
charter school personnel lacked the time and resources to get involved in meaningful 
collaboration. The theme that running a charter school is an all-consuming occupation 
appears in many charter-school evaluations. 

Wells did find that charters had more programmatic offerings from which parents 
could choose. She also observed, however, that charters were doing some of the 
choosing. Few of the charters provided transportation. Thus only parents with access to 
a vehicle and either no job or a flexible schedule could choose charters as an option. Few 
of the working poor have even minimal flexibility at the workplace. 41 

The UCLA group found mixed results for autonomy. Administrators did report 
that their area of greatest importance was the area in which they had greatest autonomy: 
teacher hiring. 

In California, virtually all charters are chartered by, and are part of, the local 
school district. In some other states, they are chartered by other entities. In Michigan, 
for instance, most are chartered through state universities and are considered as 
individual Local Educational Agencies. One might well expect to find more feelings of 
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autonomy in such instances. In Michigan, too, though, three-fourths of the charters are 



operated by Educational Management Organizations (EMOs), and this might well lead to 
reduced perceptions of autonomy as EMOs bring with them specific curricula and 
instructional methods that leave teachers little room for decision making or innovation. 
(Michigan’s charter schools will be addressed more fully later in this report.) 

More recent charter school evaluations in California have been able to look at 
achievement data. In the most precise and widest in scope of these, charter school 
students are progressing about the same as those in public schools. ' A 2003 study by 
The RAND Corporation differentiated three types of schools: conventional, conversion, 
and start-up. Conversion schools were conventional schools that converted to charter 
status, while start-ups built charter schools from scratch. For the most part, in terms of 
test scores, the three types of schools look quite similar at the elementary level — and 
charters are overwhelmingly elementary schools. At the secondary level, start-ups have a 
small edge over both conventional and conversion schools in reading and math. 

When the study examined the results for schools that offered instruction outside 
of regular classrooms, it found such schools scored much lower. As the authors note, 
however, the study did not control for selection, and students in non-classroom oriented 
schools might differ from others on important variables. 

The researchers were able to find a group of six districts that had the capacity to 
track individual students over time. The test data from this study led the researchers to 
conclude “charter school students are keeping pace with comparable students in 
conventional schools.” 43 
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That students in California charters are “keeping pace” with students in 



conventional schools has been cited by the Center for Education Reform as indicating 
that the RAND study is one of “88 major studies” favoring charters. 44 The lead author of 
the study, Ron Zimmer, was quoted in the Los Angeles Times as saying “Charter schools 
may not necessarily be the silver bullet in terms of solving all education woes, but they 
seem to be a concept that is worth pursuing.” 45 The New York Times portrayed the 
RAND study as finding that California charter schools “typically perform as well as their 
conventional counterparts even though they are much less likely to receive money in 
certain major categories of educational spending and rely far more often on uncertified 
teachers.” 46 

Given charter advocates’ promise to raise achievement, however, one can view 
“keeping pace” as reflecting charter school failure. This is particularly true in the case of 
California. A January 2004 PBS John Merrow documentary was called “First to Worst: 
The Rise and Fall of California’s Public Schools.” Whether or not one agrees with this 
characterization or the analysis of the program (California’s enormous demographic shift 
would have caused some decline because of the large influx of non-English speakers), it 
is true that on the 2003 NAEP fourth-grade reading assessment, California was 49th 
among the 50 states. Its score of 206 placed it one point ahead of Fouisiana’s 205 and one 
point behind Alabama’s 207 (the national public school average was 216). At the eighth- 
grade level, California tied Hawaii for last place. In the NAEP 2003 mathematics 
assessment, California did only a little better. Its public school fourth graders scored 227, 
four points ahead of Alabama, Mississippi and New Mexico. California public school 



Page 20 of 61 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0505-113-EPRU.pdf 





eighth graders scored 267, besting Mississippi by six points, Alabama by five, New 
Mexico by four and Arkansas, Hawaii, and Louisiana by one. 

Thus, if charter students in California are “keeping pace” with those in other 
public schools, they are keeping pace with the lowest-scoring students in the country. 

This is not, in theory, what was supposed to happen. School critics have expressed 
outrage over the performance of California public schools. Logically, then, one would 
expect them to express similar outrage about California’s charter schools. 

David Rogosa at Stanford University, working with data from all of California 
schools, generally affirms the RAND results. 47 Rogosa has tracked changes in the 
Academic Progress Index , although he cannot track individual students over time as 
RAND could. He is limited to looking at the scores of successive cohorts of students at 
each grade level. 

Rogosa observed gains for both charters and conventional schools. At the 
elementary level, they are very close. In grades 2-6, from 1999 through 2002, charters 
gained 94.3 points while conventional schools gained 96.3 points. 

At the secondary school levels, though, charters fare poorly. Grade-7 charter 
school students gained 17.9 points while grade-8 charter school students gain a mere .25 
points. This compares to 53.1 points, and 38.9 points in the two grades, respectively, for 
public school students. Conventional schools also outperformed charter schools in the 
grades 9 to 11 comparison, 24.9 points to 14.8 points. 

Looking at only students in schools where many students are economically 
disadvantaged, the results show a slightly higher increase for charters at the elementary 
level, but the advantage for conventional schools is greatly heightened at the middle and 
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high school levels. Thus, at the elementary level, disadvantaged charter school students 
appear to be “keeping pace” with their public school peers, but at the middle and high 
school level, charter school students are falling behind the low- scoring public school 
students. 

Another study of California charter schools reached a different conclusion than 
Rogosa. Margaret Raymond of the Hoover Institution claimed that “The striking 
finding is that their [charter high schools’] average improvement is more than twice that 
of conventional high schools.” Raymond did not find it equally striking that gains in 
charter middle schools were less than half of those in traditional schools and what 
Raymond called “competitor schools,” public schools in districts with at least one charter 
school. Moreover, her conclusion about high schools resulted from a statistical error. 
Raymond treated the high school as the unit of analysis. This gave each high school the 
same weight, as if they were the same size. This is analogous to giving Mississippi the 
same weight as New Jersey in calculating an SAT average. Mississippi has many fewer 
students and only 4 percent of its seniors take the SAT. New Jersey is much larger and 
almost 80 percent of its seniors take the test. To treat all schools as if they were the same 
size greatly increases the importance of small schools relative to large schools. 

Correcting this error leads to the results found by Rogosa. 

Texas 

[Renaissance Charter School is] a decrepit, two-story gray stucco office building 
that sat woefully along a busy commercial street. The City of Arlington had 
declared the second level of the vacant building unsuitable for habitation, so the 
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school set up shop in two large rooms on the ground floor. The building had no 
heat. The classrooms had no desks, no chairs, no textbooks, no chalkboards, no 
trashcans no gymnasium, no lunchroom, no vending machines, no functioning 
toilet, 

“If you name it, we don 7 have it, ” said one student. 49 

One wonders how such situations as Renaissance Charter School can arise in the 
first place. Renaissance was cited by the operator of another charter school as an 
example of a high-quality charter. State Board of Education member, Alma Allen, also 
gave Renaissance high marks until the true conditions came to light. “I held Renaissance 
up as a beacon light,” she said at the Board’s May 1999 meeting. “I’m very disappointed 
today.” 50 

How many other disappointments are out there, unseen by officials, particularly in 
light of a statement by a researcher at the Texas Center for Educational Research (who 
asked for anonymity)? When asked why the legislature had reduced the funding for the 
annual charter school evaluation so much that only TCER and not the usual four 
organizations could afford to participate, the researcher said, “Because they don’t want to 
see what they’d find.” 51 (The other three organizations that participated in the earlier 
evaluations were the School of Urban and Public Affairs at the University of Texas at 
Arlington, the Center for the Study of Education Reform at the University of North 
Texas, and the Center for Public Policy at the University of Houston). 

The results presented in the fifth-year and sixth-year evaluations do not favor 
charters. It is true that students who are in charters gained more over a three-year period 
in terms of percent passing the Texas Assessment of Academic Skills (TAAS) than did 
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students who remained in public schools all three years. Their rates, however, remain 
below those public school students. 

More important, the evaluation presents only pass rates, not the actual scores. 

The passing scores were set initially near the chance level, and small changes in scores 
produced large changes in pass rates. Since the gains reported were in pass rates and not 
scores, it is possible that the substantial increase in passing rates represents only a small 
gain in scores. Because they started with higher pass rates, public schools could have 
raised test scores more than charter, but these increases would not necessarily be reflected 
in pass-rate statistics. The pass rates for the public school students might have been 
muted by ceiling effects. For students in grade eight or lower, those in charters for three 
years and those in public schools for three years, Table 1 shows the pass rates for TAAS 
reading. 

Table 1: TAAS Reading Pass Rates for Students Grade Eight or Lower who have 
Attended Charter Schools or District Schools for Three Consecutive Years 





Charters 


District 


2000-2001 


65.0 


81.5 


2001-2002 


78.1 


89.5 


2002-2003 


85.1 


92.0 



Source: Texas Center for Educational Research. (2003, July) Texas Open-Enrollment Charter Schools: 
Sixth Year Evaluation. 

The trend for mathematics is similar. Trends for those who went back and forth 
between publics and charters over the three-year span show no discernible pattern. The 
Texas Education Agency in 2001 identified “peer schools” — public schools matched to 
particular charter schools on demographic characteristics. For 2001-2002 and 2002-2003 
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the peer public schools had higher pass rates on both TAAS reading and math. They had 
higher attendance rates and lower dropout rates. The Houston dropout rate debacle has 
called the state’s dropout records into question; however, some schools in Houston with 
more than 1,000 ninth graders had fewer than 300 12 th graders, and zero dropouts. 52 

These results hold both for schools that serve fewer than 70 percent at-risk 
students and for schools where more than 70 percent of the students are at risk. 

In terms of systemic impact, only 54 percent of the public school officials 
interviewed by TCER were aware of the presence of charters, even though all such 
officials were in public school districts that contained a charter school. Under such 
conditions, it is no surprise that TCER found that “charter schools have had little impact 
on educational approaches and practices of traditional public schools.” 

The Executive Summary for the Sixth Evaluation provides a concise conclusion: 
Across six school years, traditional public schools have outperformed charter 
schools. Only 14 percent of charter schools earned the highest Texas 
accountability ratings in 2001-2002 (Exemplary, Recognized, Commended), 
while 86 percent received one of the lower ratings. Furthermore, an increasing 
percentage of charter schools have applied for ratings under the less academically 
rigorous alternative educational accountability system. Even when more 
equitable comparisons are made with peer campuses with similar enrollments, 
school characteristics, and student demographics, dropout rates are higher and 
TAAS passing and attendance rates are lower for charter schools. 

The Seventh Evaluation is not yet available. Kelly Shapley, director of the 
TCER, reports that the results generally accord with the Fifth and Sixth Evaluations, but 
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some factors are unknown because of a shift in the test used, from the TAAS to the 



apparently much more difficult TAKS — Texas Assessment of Knowledge and Skills. In 
addition, an alternative evaluation system is being developed that will apply to about half 
of the charter schools. She did report that students in charter high schools were not doing 
as well as those in regular public schools, especially in mathematics. She believes this is 
due largely to differences in teacher quality: Teachers in Texas charter schools are not 
required to have a major or minor in the subjects they teach, nor even to have a 
bachelor’s degree. 54 

In 2003-2004, 43 percent of charters were not included in the state system of 
school accountability, but they will be in 2004-2005. There are currently allegations of 
“student dumping.” In a four-month period leading up to the state’s testing in February 
2005, over 400 students moved from charter schools to traditional public schools. “This 
figure appear inflated to me, and it is almost as if students are being dumped off onto 
school districts for the sake of ratings,” said state Senator Mario Gallegos, who asked 
Education Commissioner, Shirley Neeley, for an investigation. 55 It is too early to say 
whether this number exceeds the normal flow of students from charters to public schools 
because the students are dissatisfied with the charter or because the charter has counseled 
them back to a public school. 



Ohio 



The atmosphere surrounding the development of charter schools in Ohio — called 
“community schools” there — was described in 1999 by Akron Beacon-Journal 
investigative reporters Doug Oplinger and Dennis Willard. They noted even then that 
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for-profit managers of charters were “making strong inroads” while oversight was 
virtually non-existent because members of the Ohio Board of Education “say they have 
almost no authority to reject” a charter school proposal. They also observed that a 
belatedly established oversight office was undermanned. They continued: 

Most charter schools are not models for reform. First year scores indicate that 
students in charter schools are doing dramatically worse than public school 
children, and the new schools are not incubators for innovation as proponents 
promised they would be. 

Profits are being reaped, but there is no evidence that charter schools are 
reducing education costs or saving Ohio taxpayers money — despite lower pay for 
teachers and exemptions from 191 state mandates that hike the costs of education 
in public schools. 56 

Seven years later, little has changed. Ohio’s charter schools reap over $400 
million annually. The “undermanned” oversight office mentioned by Oplinger and 
Willard is the Legislative Office of Education Oversight (LOEO). While Oplinger 
indicated in a personal communication in 2001 that the legislature had a history of hinting 
it would do away with the LOEO whenever the agency offended the body’s faith in 
charters, the oversight agency has nonetheless pressed on. The LOEO has issued five 
annual reports on charter schools in Ohio; its fifth and final one, issued in 2003, reflects 

co 

deep frustration at the non-cooperation from the charters. 

When LOEO compared achievement at traditional schools with community 
(charter) schools, 14 of 20 comparisons were statistically significant and 13 of those 14 
favored traditional schools, but the effect sizes from the comparisons were small. When 
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LOEO compared each community school to its matched traditional school, it called the 
outcomes “mixed.” Of the 415 comparisons, 270 were not significant. Of the 145 that 
were significant, 103 favored traditional schools, 42 favored community schools. When 
the comparisons were made using scaled scores from the Ohio Proficiency Tests, the 
effect sizes were small. When the comparisons involved passing rates on those tests, 
though, the effect sizes were large and favored traditional schools. 

LOEO had to eliminate 28 percent of the community school results, however, 
either because the community schools did not submit the results at all or because it had 
serious concerns about the quality of the test data they did submit. LOEO had even less 
success comparing the results of the proficiency tests to the test standards in the charter 
school contracts. Only 17 of 50 schools provided useable data. Those 17 schools met 
only 39 percent of the proficiency test goals stated in contracts with their sponsors. 

The LOEO summed up its findings thusly: 

In sum, the most that can be said about the academic performance of community 
schools as a group, is that they are doing no better than low-performing traditional 
schools with similar demographic characteristics. While most community schools 
are not meeting state academic standards, many are not reporting data that allow 
them to be compared to their contracts. Those that do report data are generally 
not meeting the academic performance goals specified in their contracts. 59 

The LOEO specifically did not accuse charters of deliberately withholding the 
required data — data routinely submitted by traditional schools — but the suggestion is 
implicit. The report concludes that the legislature should continue the community school 
initiative “only” if the Ohio Department of Education and the community school sponsors 
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meet seven conditions. 60 Otherwise, the report said, the legislature should terminate 
funding. 



It is doubtful that the legislature will act on the recommendations to LOEO’s 
satisfaction. It seems generally favorably disposed to choice in various forms. For 
instance, in its 2003 session, the legislature voted to vastly increase the Cleveland 
voucher program just after an evaluation of that program showed that in the first four 
years, the public school children had gained more on tests than had their voucher-using 
peers. 61 A more recent report by outside researchers adds a fifth year and concluded that 
while the voucher students started out ahead of the public school students, there were at 
the end of the period no differences in tests of reading, language and mathematics. 62 

While legislative action based on LOEO’s recommendations is unlikely, it is 
possible that more study of charters will occur. As Willard and Oplinger wrote in the 
Akron Beacon Journal in March 2005, “With the price tag rising and the results 
uncertain,” a pair of Republican lawmakers have introduced legislation seeking “more 
accountability and a comprehensive study of Ohio’s burgeoning charter movement.” 63 
Although the Ohio Education Association released data indicating poor performance 
continues in Ohio charters, 64 the principal reason for review is fiscal: because most 
charters in Ohio are run by for-profit EMOs, the amount of money siphoned away from 
public schools is substantial. In Dayton, where charters have their greatest popularity, 26 
percent of all students attend them, costing the Dayton school district $41 million of its 
$113 million in state aid. 65 
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Michigan 



Michigan’s charter schools are perhaps the most heavily scrutinized in the 
country. Educational evaluators, economists, and political scientists have examined 
various aspects of them. 66 Although the investigators come from a variety of professions, 
their conclusions have been remarkably similar. The overall picture was well summed in 
2002 up by Miron and Nelson: 

We found considerable variation among charter schools, with some clearly 
outgaining their host districts and others lagging far behind. In the aggregate, 
however, our findings cast doubt on proponents’ claims that Michigan charter 
schools will leverage gains in student achievement. With the exception of Grade 
4 math, MEAP [Michigan Education Assessment Program] pass rates in the 
typical charter school grew less (or fell faster) than those in their host districts. 
Finally, we found that, while there are some variations among companies, as a 
group, charter schools managed by for-profit EMOs gain less (or fell faster) than 
other charter schools. This casts at least some doubt on privatization advocates’ 
claims that introducing competitive pressures into educational management will 
lead to improvements in performance. 67 

The statement about EMOs adds a new wrinkle to what has been written earlier. 
Other state evaluations have not distinguished among forms of management, public and 
private. In Michigan, though, over a four-year period, the proportion of charters 
managed by private, for-profit firms grew from 16 percent to 71 percent, which made 
such an analysis possible. Miron and Nelson note: “Whether the costs of competition are 
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justified depends on whether the need to compete makes schools spend their other 
resources more wisely and efficiently.” 68 There was some evidence that the need for 
marketing took time and other resources away from educational activities. 

Mintrom had earlier observed that the public schools showed little interest in 
competing with the charters, a fact he ascribed to charter advocates having “often been 
unnecessarily harsh in their disparagement of public schools.” 69 

Similarly, none of the investigators found much innovation taking place in 
charters. “More striking is the degree of similarity that we find across all schools, be they 
charter or traditional,” said Mintrom. Mintrom asked principals in schools to rank 
various practices in terms of their degree of innovativeness without telling the principals 
what kind of schools had listed the practice as an innovation. He concluded that “while 
some charter schools are definitely doing some innovative things, overall Michigan’s 
charter schools are no more remarkable than many traditional public schools in their 
practices.” The charters exceeded the publics in the adoption of innovations in only 
three areas: non-traditional scheduling, mandatory foreign languages, and uniforms. 72 

Ilinois 

An evaluation of charter schools by Christopher Nelson and Gary Miron was 
delivered to the Illinois State Board of Education in July 2002. 73 As of March 2005, the 
report had not been presented to the Board and the Board had yet to accept the report or 
release it. It was obtained through a Freedom of Information Act request by the 
American Federation of Teachers. 74 In the first year of administration (1998-99), charter 
school students passed the Illinois Student Achievement Test (ISAT) at a 39 percent rate, 
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and the rate hovered around 40 percent in the ensuing two years of the study. The pass 



rate for all Illinois students was around 60 percent in all three years. 

When Illinois switched to the Prairie State Achievement Test in 2000, Nelson and 
Miron were able to compare charter results with demographically similar public schools. 
The differences on all five tests (reading, math, writing, science, and social science) 
consistently favored public schools but were small. In interviews. Nelson and Miron 
found charter school teachers and administrators disparaging of the standardized tests, 
and most charters used alternative assessments of some kind. It is possible that the 
charters’ lack of interest in standardized tests contributed to their lower performance. 
Although the study ended with the 2000-2001 school year, no further evaluation of 
Illinois charters has been commissioned. 

Hoxby and Rockoff examined charter school and public school achievement in 
Chicago. 75 Hoxby and Rockoff claimed to have compared the charter schools with the 
nearest neighborhood schools and the nearest neighborhood schools with similar ethnic 
composition. They provided no background information on students in those schools. 

An examination of the research by Nelson and Miller, though, revealed differences 
between the charters and the public schools that could produce significant differences in 
achievement. 76 To begin with, the study did not evaluate all charters but only a sample of 
nine. Secondly, the students in charter schools often lived some distance from the school. 
At one of the nine charters examined, the median distance students traveled was five 
miles. Both these students and their parents might have different levels of motivation 
than students in neighborhood schools. 
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In addition, the Nelson and Miller use of Mapquest to identify neighborhood 
schools often found schools closer than the school identified by Hoxby and Rockoff. In a 
number of instances, these neighborhood schools had higher achievement than the 
neighborhood schools in Hoxby’ s sample. On average though, the neighborhood schools 
had a much higher rate of students eligible for free and reduced price lunch than did the 
charters. 

Nelson and Miller found other odd aspects of the Hoxby and Rockoff study. For 
example, in one instance, Hoxby and Rockoff compared the average of five charter 
schools, schools authorized under a single charter but operated by five different 
management companies, to a single neighborhood school far from any of the five. No 
justification was provided for this comparison. 

North Carolina 

As noted earlier, at the time it was conducted, the Goldwater Institute study was 
alone in tracking the same students for more than one year of their academic careers. 

Most studies still provide only a “snapshot” for one point in time, or multiple snapshots 
of successive cohorts of students passing through the same grade. A 2004 study by Ladd 
and Bifulco, however, measured achievement longitudinally in charters and district 
schools. 77 Some students went from one type of school to the other and back again, and 
Ladd and Bifulco acquired measures of achievement knowing which kind of school they 
were in. Thus the students served as their own controls. This rendered unnecessary any 
measurement of background factors to ensure that the two groups were similar: “Because 
we compare the test score gains made by students while they are in charter schools to the 
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gains those same students made in public schools, these findings cannot be explained 
away by differences in student backgrounds.” 78 The study indicated that students in 
charter schools gained less than when they were in regular public schools and that these 
lesser gains were substantial: 

The negative effects of attending a charter school are large. Charter school 
students exhibit gains nearly 0.10 standard deviations smaller in reading and 0.16 
standard deviations smaller in math, on average, than the gains those same 
students had when they were in traditional public schools. Assuming such losses 
compounded annually, students enrolled in charter schools for 5 years would 
score nearly one-half of a standard deviation lower in reading and nearly eight- 
tenths of a standard deviation lower in math than they would if they remained in 
traditional public schools. 79 



National Studies 

When one turns from state-level to national studies, little changes in the 
perspective that charters offer. One report from SRI International was called A Decade of 
Public Charter Schools , but it presented no achievement data. It did mention that 
charters were usually terminated for financial reasons, not academic ones, and that those 
who oversaw charters often imposed lesser sanctions than termination for those same 
financial reasons. 80 

Similarly, a national study by the University of Washington on charter school 
accountability did not report test scores, but only what kinds of tests were used, how 
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often students were tested, and what were the relationships among the various players in 
the charter school arena (authorizers, local boards, state board, school operators, etc.). 

The report did suggest that little accountability vis a vis student achievement was taking 
place: 

Some charter schools have managed to produce acceptable student outcomes but 
are still a long way from creating strong internal accountability arrangements and 
stable working relationships between the governing boards and management. 

Only a few of the hundreds of legally designated charter authorizing 
agencies have faced their own responsibilities in holding charter schools 
accountable. Some authorizers fall back on process and compliance monitoring 
and avoid acting on measures of student performance, rather than opening 
themselves up to the criticism that they will not close or replace any of the 

o 1 

conventional schools for which they are also responsible. 

The various studies compiled by RPP International for the U.S. Department of 
Education described ways in which schools were said to be held accountable, but made 

09 

no attempt to determine if accountability policies were being enforced. 

A generally positive study of charters was released in July 2003 from the 
Manhattan Institute. This study, too, notes that “Charter schools. . .are a widespread but 
poorly-studied form of education reform.” To these researchers, this is so because “many 
charter schools serve specifically targeted populations such as at-risk students, disabled 
students, and juvenile delinquents. This makes it very difficult for researchers to draw a 
fair comparison between charter schools and regular public schools.” This is not wholly 
true, however; California, Michigan, and Ohio, studies have compared demographically 
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similar charter and traditional schools. Still, the authors’ review a few studies and 
conclude, “It should be obvious that there is very little we can say with confidence about 
the academic performance of charter schools based on this research.” (p. 5) 

The Manhattan Institute began with 1 1 states thought to have sufficient numbers 
of non-targeted charters, but had to drop four for state-level results because of insufficient 
data (the researchers do not specify the criteria for “insufficient”). Charter schools 
serving large proportions of at-risk students were kept in the study if public schools in 
that geographic area also served at-risk students. They excluded “cyber schools” and 
charters that had converted from traditional school status. They did not feel that these 
latter schools were truly schools of choice. 

What the researchers call a “national charter school effect” — one based on 1 1 
states — showed in the form of an effect size of +0.08 in math and +0.04 in reading. Such 
an effect size would move a student from the 50 th to the 53 rd percentile in math and from 
the 50 th to the 52 nd in reading. These effects are small, but they are statistically 
significant because of the huge sample sizes. 

The Manhattan Institute explained why it dropped four states from the state-level 
analysis (insufficient data, although they do not provide any criteria for making that 
judgment), but not why they dropped two others, Michigan and Ohio. If Ohio’s 
Legislative Office of Education Oversight report is accurate and Ohio charters are not 
reporting data, they perhaps could not get sufficient data in that state, but the Manhattan 
Institute gives no reason for the exclusion of Ohio and Michigan. 

At the state level, only Texas showed positive effects for charters, an outcome 
that would seem to contradict, or be contradicted by, the accountability figures from the 
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Texas Education Agency and the Texas Center for Educational Research. Arizona, 
California and North Carolina all showed effect sizes very close to zero. Florida 
displayed a peculiar outcome: for reading the effect was significant for the state’s FCAT 
reading test, but not for the nationally normed Stanford 9 reading test. For math, the 
effect was significant for the Stanford math test, but not the FCAT math test. 

The researchers’ claim of their national results: “Because these results are 
statistically significant, we can be very confident that the charter schools in our study did 
have a positive effect on test scores.” The confidence of the authors might be displaced 
because a statistically significant result could be practically meaningless. In addition, 
since the level of significance for most differences was at the .05 level, the claim has a 1- 
in-20 chance of actually being wrong. 

Caroline Hoxby at Harvard has also conducted an evaluation of charters at the 
national level. She obtained lists of charter schools and their locations from all 50 
states. She then compared the percent of charter school fourth graders who scored 
“proficient” or better on the state’s tests of reading and mathematics with the percent who 
scored proficient or better in the physically nearest public schools as well as the 
physically nearest public school that had the same ethnic mix as the charter school. If a 
state did not test in fourth grade, third- or fifth-grade students were used. Not all states 
had sufficient numbers of charter students to generate meaningful state-level data: 
California had over 1 1,000 fourth grade charter schools students, while Virginia and 
Wyoming had 14 each. For the 21 states that produced meaningful data, 3.8 percent 
more charter school students were proficient in reading and 1.6 percent more were 
proficient in math. When compared with the nearest school with a similar racial 
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composition, the differences were 4.9 percent and 2.8 percent, respectively. As noted in 



the discussion of Hoxby’s Chicago study, the “nearest neighborhood school” procedure 
appears to be methodologically flawed. 

These differences are not large as they stand, but Hoxby made at least one error 
that would reduce the difference. For the District of Columbia, for charter school 
students, she used the definition of proficient the District reported for NCLB 
requirements, but for the public school students, she used the test maker’s much more 
rigorous definition of proficient. This resulted in an apparent difference of about 40 
percentage points favoring charter schools students. This difference looked impossibly 
large in light of a Washington Post article that found charters generally scoring lower 

Of 

than public schools. In a follow up paper, Hoxby corrected for this deficiency, but 
deleted some charters that were self-described as serving at-risk students. 86 No rationale 
was provided. In neither paper did Hoxby provide any discussion of why her results, 
which generally favored charters, were contradicted by the many state-level analyses 
discussed earlier in this paper. 

Doubts from a Variety of Sources 

A somewhat skeptical report from Western Michigan University was titled 

87 

Student Achievement in Charter Schools: What We Know and Why We Know So Little. 
Among the “whys are some legitimate reasons: new schools, too few schools to merit a 
statewide evaluation, small schools where evaluations might collide with privacy laws, 
schools operating in grades not tested by the state, and changes in state testing programs.” 



Page 38 of 61 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0505-113-EPRU.pdf 





In some states, the schools apparently collect the data but do not report it to a 



central source or, as was noted in Ohio, do not report it accurately. Miron and Nelson 
report: “We have sought in many of our own statewide evaluations to collect norm- 
referenced test data directly from schools. However, a combination of reluctance on the 
part of charter school officials and the sheer scale of the task yielded little usable data.” 
One final reason cited by Miron and Nelson simply involves decisions not to 
collect data: 

Finally, political factors may discourage state officials and others from 
commissioning, sponsoring, and funding statewide evaluations of charter school 
achievement.... high-profile, statewide evaluations of charter schools’ impacts on 
student achievement may appear threatening to policymakers since unexpected 

oo 

findings might diminish the credibility of popular policy. 

Such a statement echoes an earlier comment from Manno: 

Today, it’s hard to know how well charter schools are actually doing... One 
reason for the dearth of good accountability information lies with charter 
authorizers and operators. Truth be told, they are often content to leave 
accountability agreements nebulous and undefined. Leaving accountability 
agreements vague and indeterminate is fraught with danger because over the long 
term this approach is more likely to lead to a charter school being subjected to the 
rule- and compliance-based accountability practices that characterize 

on 

conventional schools. 
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In 2005, it’s still difficult, but the bulk of the evidence leads to the conclusion that 



the charter schools, in general, are not delivering on their promise to improve 
achievement. Held up against the Nathan Criterion, though, charter schools are found 
wanting. 

Not delivering on the promise is the more surprising because charter schools have 
the twin advantages of being small with small classes. Research indicates that both small 
schools and small classes serve to increase achievement. 90 

Katrina Bulkley has suggested that a promise along the lines of Joe Nathan’s 
should not have been made since it could not be lived up to. 91 Bulkley, citing Lake and 
Millot’s 1998 treatise on charter school legislation, " concluded that “performance 
accountability for charter schools rests on two key assumptions: 1) Authorizers can 
assess the quality of education offered by charter schools using test scores and, if needed, 
other methods, and 2) authorizers will act on their assessments by revoking or not 
renewing charters that do not demonstrate that they are providing quality education.” 

The studies reviewed in the earlier sections of this paper suggest that the first 
assumption usually holds, but the second does not. Sometimes the lack of action is due to 
a lack of manpower by the oversight agency, a lack of data provided by the charter 
schools, and a lack of political will. In her review of the data on accountability to date 
(then early 2001) and her interviews with charter school authorizers, Bulkley distilled a 
“constellation of challenges” to authorizers that together constitute what she labeled the 
“accountability bind:” 93 

• Educational performance is not simple to define or measure, nor is how good 
is “good enough” in educational quality. 
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• Other aspects of a school’s program, often more difficult to measure than test 
scores are also important to families and authorizers. In this context, 
authorizers sometimes turn to “proxies” to assess school quality. 

• Teachers, parents and students become highly invested in particular schools 
and destroying a community may be more difficult for authorizers than 
serving a diffuse public interest. 

• Charter schools have become a highly politicized issue on both sides and 
some authorizers are concerned about their decisions reflecting poorly on 
charter schools as a reform idea. 94 

It is easy to empathize with these all-too-human outcomes. At the same time, 
however, it must be said that it was the “diffuse public interest” that created charter 
schools in the first place. 

Bulkley suggested a “middle ground.” She noted that when public schools are 
found lacking, the schools are not generally closed. Bulkley felt that the forces operating 
against closure will result in many charter schools having their charters renewed. The 
four years since the Bulkley’ s paper first appeared have proven her correct in that. 
Bulkley’ s “middle ground” approach calls for something less than the Nathan Criterion: 

The way in which charter advocates have presented the idea of a performance 
contract is, fundamentally, as an “all or nothing” proposition. In arguing for a 
broad-based system of contracting, Hill and his colleagues make the case that 
“Contractors who failed to provide instruction as promised, or whose students 
were low and not improving as anticipated, could be fired or given an ultimatum 
to improve or be replaced.” In addition, they argue that “accountability only 
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means something if there are consequences when the children do not learn.” The 
rationale behind the additional autonomy provided to charter schools is that there 
will be real and significant consequences if performance is not demonstrated. 

Some might say that the No Child Left Behind Act (NCLB) has made the issue 
moot because that law leads to significant consequences for all schools if performance is 
not demonstrated. Some also say that the requirements of NCLB are unreasonable 95 or 
even irrational to the point of being unconstitutional. 96 Nevertheless, NCLB will evaluate 
charters and public schools with a common metric, Adequate Yearly Progress (AYP), 
within each state. The question will then be: Do larger or smaller proportions of charter 
schools make AYP than demographic ally comparable public schools? 

Summary and Conclusion from State and National Studies 

Amy Stuart Wells summarized the situation of charters in 2002: 

Thus far, there is no strong or consistent evidence that charter schools have 
improved student achievement or that they are being held more accountable for 
academic outcomes than regular public schools. Aside from anecdotal reports 
from individual schools, none of the methodologically sound state-level reports 
show significant increases in overall achievement of charter school students, and 
many show decreases. In fact, the lack of academic or outcome-based 
accountability in charter schools is perhaps one of the most robust findings across 
the states and reports. 97 
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A string of negative headlines has not, as yet, led to any reconsideration of charter 
laws. Indeed, charters continue to expand and in June 2004, the U.S. Department of 
Education awarded California $75 mi llion for new charters. Thus, the movement 
expands in the absence of accountability. How did this happen? Wells believes that 
while early declaimers on charters clearly intended to honor The Nathan Criterion, others 
came along who had no such intention. For them, deregulation was an end in itself: 

Clearly, it is difficult to read the emerging body of research literature on charter 
schools and not come to the conclusion that the free-market reformers won the 
battle for the soul of a movement that promised to be so much more than merely a 
deregulatory reform. In other words, despite the diversity of people and political 
interests represented in this reform, the legislative agenda in the statehouses 
across the country has been dominated by those who want deregulation for the 
sake of competition and school-level autonomy. The interests of those who 
would like to see greater accountability in exchange for that autonomy, or those 
who would like to have gained more meaningful community control of schools in 
low-income communities, have not been at the forefront of the policy-making 
agenda. 98 

At the present time, the federal government is actively pursuing a de -regulatory 
agenda generally, giving such policies more prominence. The recent series of corporate 
CEOs and Wall Street analysts charged with various felonies might lead at least some 
people to rethink their faith in the market and competition alone as unvarnished goods. 
Corporations are authoritarian, even totalitarian, organizations, not democracies. 
Democracy is messy. The desire for the efficiency bragged about by corporations is, 
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when applied to social systems, a desire for totalitarianism. For Wells, if democratic 
thinking cannot be applied to charters, then the charter school movement as a means to 
improved education is dead: 

Thus the only remaining hope for the charter school movement to have any 
lasting positive impact on the public education agenda would be for the more 
progressive members of this diverse and complex movement to recapture the 
language and symbols of what constitutes a good charter school law. Until that 
happens, the hopes and dreams of thousands of social justice educators and 
families engaged in this reform will be marginalized and reliant on powerful 
private market agents who have never served the most disadvantaged students 
well. It is time to leave the market metaphor to the market and to focus the 
educational policy lens on equal opportunities and the very hard work of teaching 
all students well." 



The Little-Dealt-with Issue of Attrition 

Although charter schools do not perform any better than public schools, if as well, 
attrition rates tend to confound any analysis of trends in their test scores — in some cases 
boosting test scores, in other cases lowering them. The attrition rate is the rate at which 
students leave a school. In the section on Arizona, it was noted that the Goldwater 
Institute analysis benefited from the high attrition rate of lower scoring students returning 
to public schools. By contrast, in the North Carolina section, it was reported that 
researchers felt a substantial factor in charter school low performance was the high 
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attrition. Although attrition rates declined with each year that the school had been open, 



after five years charter schools still had a rate that was twice that of public schools. In 
the case of North Carolina the mobility rate itself, measured over a period of five years, 
appeared to affect the students' scores. High mobility rates are often cited as causes of 
both lower achievement and later dropping out, especially if the change in schools is not 
due to a change in residence. 100 

Aside from these studies, most information about charter school attrition is more 
anecdotal. There were stories that Edison Schools, Inc., would “counsel out” low scoring 
students or students who might need special education services. The 2005 book, The 
Charter School Dust-up observes, “Charter schools may find it easier to push out students 
who are not performing well, and this de-selection could depress public school score 
levels in the year after a student leaves a charter school.” 101 The same process would, of 
course, raise the score level of the charter school itself. As noted there are allegations 
that charters might be “dumping” students as the time for state or NCLB testing 
approaches. 

Teacher attrition is also an issue that has not received much attention. Stories 
occur about high teacher attrition rates, but they usually apply only to one school and 
occur most often in stories about how teacher attrition is reflective of or contributive to 
problems at a given charter. However, a recent analysis by the Austin American- 
Statesman of data in a report from the Texas Center for Educational Research led to the 
conclusion, “The average Texas charter school loses almost half its teachers every year 
and replaces the lost staff with teachers who are less experienced and have less college 

i m 

education than those working in other public schools.” 
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The Statesman found that the turnover in 2003-2004 was 44 percent and that the 
rate had varied between 40 percent and 55 percent per year over a five year period. For 
all Texas public schools, including the 200+ charters, the figure was 14 percent for 2003- 
2004. The Statesman cited Richard Ingersoll of the University of Pennsylvania as saying 
that the typical public school loses 15 to 25 percent per year and that any school that 
loses a third or more of its teachers bears serious scrutiny. Ingersoll specializes in issues 

1 0T 

surrounding teacher preparation and retention. 

The high turnover rate has resulted in Texas charters having a teacher core where 
1 1 percent of teachers lack a bachelor’s degree and 73 percent of teachers have five years 
or less of experience. This compares to statewide averages of 1 percent and 36 percent, 
respectively. 104 



Charter School, Politics, and Ideology: Reactions to Two 

Federal Analyses 

The previous sections have shown that accountability for charter schools is weak 
and that, when charters are held to account, it is typically for some reason other than low 
academic performance. Events in summer and fall 2004 suggest that those favoring 
charter schools will resist any data that contradict an ideological faith in charters. 

As part of the 2003 National Assessment of Educational Progress (NAEP) reading 
and mathematics assessment, charter schools were sampled as a separate group. By fall 
of 2003, the U. S. Department of Education had posted the regular NAEP assessment 

Page 46 of 61 

This document is available on the Education Policy Studies Laboratory website at: 

http://www.asu.edu/educ/epsl/EPRU/documents/EPSL-0505-113-EPRU.pdf 





results, but had not moved to report the charter school data. By summer, 2004, no 
analysis was yet forthcoming. Researchers at the American Federation of Teachers 
(AFT) located the raw data on the NAEP site and analyzed them. Overall, the public 
schools outperformed the charters. Children eligible for free and reduced price lunches in 
public schools scored higher, as did children in public schools in central cities. When 
analyzed by ethnicity, there were no differences between charter and public schools, but 
the achievement gap in charters was as large as in regular public schools. 105 

On Tuesday, August 17, 2004, the New York Times published the AFT results on 
page 1 under the headline “Nation’s Charter Schools Lagging Behind, U. S. Test Scores 
Reveal.” 106 On Wednesday, a Times editorial called the results a “devastating setback for 

107 

No Child Left Behind, which authorizes failing schools to convert to charter status.” 

Such conversion would be available for schools that suffered five consecutive years of 
failing to make Adequate Yearly Progress (AYP). It was anticipated by many that such 
conversions would be popular, but the NAEP data called the meaningfulness of such 
conversions into question. 

Conservative charter supporters reacted swiftly and with condemnation — of the 
newspaper and of the study. The usual medium of protest, the letter to the editor, was 
ignored. Before Tuesday gave way to Wednesday, Andrew Rotherham, who writes the 
Weblog (www.eduwonk.com) for the Progressive Policy Institute (a centrist Democratic 
think tank), had penned “Live by the Sword, Die by the Times. ” On Wednesday, the 
Wall Street Journal carried an op-ed from William Howell, Paul Peterson, and Martin 
West of Harvard, “Dog Eats ALT Homework.” The tone of ridicule that op-ed struck 
appeared in other places. Lor instance, the editors of the Chicago Tribune called the 
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findings “as new as a lava lamp, as revelatory as an old sock and as significant as a 
belch.” 109 Also on Wednesday, a second Times article was largely devoted to a defense 
of charter schools by Secretary of Education, Rod Paige. 1 10 

The original Times piece had quoted former assistant secretary of education, 
Chester Finn, as saying, “The scores are low, dismayingly low. A little more tough love 
is needed for these schools.” Twenty-four hours later, though, Finn writing in the New 
York Post, declared himself incensed by the study: “This week’s firestorm over the 
performance of charter schools can be traced to a mischief bearing grenade hand- 
delivered by the charter hating American Federation of Teachers to the New York 
Times.” 111 

Finn’s use of the word “firestorm” is telling — aside from Rotherham’s blog, no 
public reaction had yet occurred when Finn’s op-ed appeared. His was part of the first 
salvo that struck on Wednesday. Thus, the firestorm metaphor must refer to the level of 
phone, fax, and email activity within the circle of charter partisans who would attack the 
article and the analysis. 

Thursday found the Reverend Floyd Flake defending charters on the op-ed page 
of the Times. " The Times identified Flake as a former New York Congressman, but 
failed to also identify him as the President of the Charter Schools Division of Edison 
Schools, Inc. The Manhattan Institute’s Jay P. Greene called the study “sheer nonsense” 
in an op-ed in the New York Sun. Jeanne Allen, President of the Center for Education 
Reform, and the most zealous of the charter school advocates debated a principal author 
of the AFT report, Bella Rosenberg, on National Public Radio’s Tavis Smiley Show, 
while the U.S. Department of Education sent former Heritage Foundation analyst Nina 
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Shokraii Rees to do the same on “The News Hour With Jim Lehrer.” Eduwonk’s 
Rotherham continued his commentary with headlines such as “Charter Cheap Shot, Day 
III,” referring to the AFT analysis as “nefarious.” Perhaps the greatest overreaction to the 
single Times article came from the Center for Education Reform, where Allen wrote, 

“The AFT has been working on their plan for months to twist NAEP data and attack the 
nation’s unsuspecting 3,000 charter schools with a full-force media blitz.” The New York 
Post, having carried an op-ed by Finn the previous day, weighed in with its own editorial, 
“Kids Come Fast.” It stated, “The AFT hates them (charter schools) because they 
threaten the union’s public school monopoly.” 114 

Neither Finn nor Post editors appeared to realize that without their endorsement in 
1988 by AFT president Albert Shanker, there likely would not be any charter schools. 

The power of the Times story and subsequent editorial were heightened by events 
elsewhere. The day prior to the article, the Los Angeles Times had reported the sudden 
closure of 60 charter schools that left hundreds of teachers and administrators with no 
jobs and left 10,000 students with no school three weeks prior to the start of the school 
year. A number of observers had worried that the additional autonomy of charters could 
lead to increased risk of fraud and other malfeasance; the article described criminal 
investigations into the finances and activities of the charters’ operator. 115 The New York 
Times article referenced the Los Angeles Times’ story. Coming on the heels of an article 
that could be taken to reflect charters’ lack of concern about their students’ welfare, the 
New York Times piece could have carried more weight than if it had appeared alone. 

The reaction continued. Perhaps to add some technical heft to the complaints, 
Robert Ferner, acting Commissioner of Statistics for the National Center for Education 
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Statistics, a division of the U. S. Department of Education, published a letter in the New 
York Times claiming that the “report is not a true picture of what is happening 
nationally.” 116 Why the National Assessment of Educational Progress would not give a 
true national picture, and what knowledge he possessed to know that its national picture 
was false, Lerner did not explain. Lerner also pointed out that the number of background 
factors in NAEP was limited and those that were present in the database could not be 
accounted for simultaneously. This is true. The irony is that these limits on the NAEP 
database were established by Finn when he was Chairman of the National Assessment 
Governing Board. Finn feared that adjusting for background variables could lead to 
excuses being made for poor scores. The limits on the NAEP database, however, had 
not stopped NAEP officials and others from making pronouncements over the years on 
the condition of public education that were identical to those about charters contained in 
the New York Times. 

The most eye-catching reaction to the article occurred on August 25, when a 
group of 31 academics, purportedly assembled by Harvard’s Peterson, took out a full 
page advertisement in the New York Times to critique both the study and the Times ’ 
coverage of it. Allen’s Center for Education Reform bore the $125,000 cost of the ad, 
which carried the headline, “Charter School Evaluation Reported by the New York Times 
Fails to Meet Professional Standards.” Two weeks later, the ad appeared in Education 
Week minus two signatories — including one Nobel Prize winner in economics— who had 
advised that they did not realize what they were getting into. The ad can be viewed at 
the Center for Education Reform’s website (http://edreform.com/_upload/New 
Y orkT ime s Ad .pdf) . 
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Among the flaws the signatories found in the Times coverage was one on 



“Journalistic Responsibility:” “The news media has [sic] an obligation to assess 
carefully any research sponsored by interest groups engaged in policy analysis. Such 
studies need to be vetted by independent scholars as in commonly done in coverage of 
research on the biological and physical sciences.” This was a curious standard 
considering that many of those who signed the ad had published, and continue to publish, 
articles that journalists cannot vet via independent scholars because only those publishing 
the study have seen it. 

Indeed, a number of signatories had published research that had not been peer 
reviewed. One of these, Hoxby, condensed a study into an op-ed for the Wall Street 
Journal shortly after the ad appeared. It is unlikely that the 31 signatories chastised the 
Journal for not vetting the piece by independent researchers (Hoxby had even more 
hastily placed the study on her website. As noted earlier, the study contained 
misinterpretations and errors of fact). 119 Lawrence Mishel, a researcher at the Economic 
Policy Institute, observed that “many of these guardians of professional standards have 
repeatedly violated the principles they now proclaim. ~ Appendix A of The Charter 
School Dust-up provides an extensive examination of how some people — Robert Lerner, 
then at the National Center for Education Statistics; former Secretary of Education Rod 
Paige; Deputy Secretary of Education Nina Shokraii Rees; Center for Education Reform 
President Jeanne Allen; Rev Floyd Flake; the Manhattan Institute’s Jay P. Greene; 
Kaleem Caire, then of the Black Alliance for Educational Options; Howard Fuller, 
Chairman of the Charter School Leadership Council; and Andrew Rotherham of the 
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Progressive Policy Institute — all used different, and lower, standards to evaluate charter 



191 

schools than they had used to evaluate public schools. 

In December 2004, The Department of Education released its own analysis of the 
NAEP charter school assessment. It showed no deviance from the AFT’s analysis. 
The Department of Education released the study via a press conference presided over by 
Darvin Winick, Chairman of the National Assessment Governing Board, and Eugene 
Hickok, then the Deputy Secretary of Education. Both emphasized the tentativeness of 
the data. Winick said, “The pilot study was a trial, however, and the need for caution in 
using the results is apparent.... Most charter schools are relatively new, and charters are 
not evenly distributed across the country. Few students have been in a charter setting for 
much of their education.” Given this comment, it was something of a surprise that the 
Department report showed that the longer the charter had been open, the lower its scores 
as indicated in Table 2. 



Table 2: Average NAEP Scores for All Charter Schools 



Years of Instruction 


Math 


Reading 


0 to 1 Year 


235 


225 


2 to 3 Years 


232 


214 


4 to 5 Years 


227 


212 


6 or More Years 


228 


210 



Source: National Center for Education Statistics, U. S. Department of Education. (2004, December). 
America ’s charter schools: Results from the NAEP pilot study. 



Only the scores for charters open 0 to one year are above the average for public 
schools, which was 234 in reading and 217 in math. Any explanation of why the trend 
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exists would be speculation, but it would not have been predicted given Winick’s 



remarks. 

While some differences were not statistically significant, of the 22 basic 
comparisons in reading and math, 20 favored public schools. Hispanic students in 
charters scored one point higher in reading while white students in charters tied white 
students in public schools. 

Both Winick and Hickok reiterated the Department’s support for charter schools. 
The author of this report was present during the question and answer session that 
followed the formal presentation of the data. A Los Angeles Times reporter, Nick 
Anderson, asked why Hickok and Winick seemed so satisfied in the near parity of 
charters and conventional public schools when the charters had promised more. Hickok 
replied that charters “are doing the same with less money.” Anderson then asked “Does 
that mean that money really does matter.” There was no answer. Hickok then 
spontaneously offered that “charter schools that don’t work don’t stay open.” This, of 
course, was contradicted by the previously discussed study that he had commissioned 
though SRI International. 123 

Perhaps the supporters and skeptics of charter schools will now reach agreement 
on what criteria they should apply to research before they publish it. 
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